Electronic image color plane reconstruction

ABSTRACT

The color of a color image having at least two sets of image pixels having color and luminance values is corrected by generating, for each color value, a low spatial frequency monochrome image including a set of smoothed image pixels. A color correction function is applied to selected sub-sets of the image pixels to generate corresponding sub-sets of corrected, smoothed image pixels. The correction includes a contribution from smoothed image pixels in one or more of the other sets of smoothed image pixels. The sub-sets of corrected smoothed image pixels is completed by interpolation or extrapolation. Each set of corrected smoothed image pixels is used to generate a corrected color image by generating one or more high spatial frequency luminance images that are combined with each of the corrected low spatial frequency monochrome images to form a full color image.

This invention relates to an image processing method and an imageprocessing device for correcting the colour of an electronic colourimage.

It is common to produce consumer colour sensors by introducing analternating pattern of colour filters onto the array of individualsensor elements. This is often referred to as a colour mosaic and acommonly used variant is the RGB Bayer pattern, which has alternatingrows of green/red and blue/green pixels (thus having twice as many greenpixels as red and blue) on a regular grid. The goal of colour planereconstruction (also called de-mosaicing) is to generate a full RGBimage from an image captured with such a sensor.

No single colour plane has the full image resolution. In terms of totalpixel count, with the Bayer pattern, green has half the resolution andred and blue each have one quarter.

It is very difficult to construct colour filters for sensors whichexactly match the spectral characteristics of our eyes or which exactlymatch the primary colours used in computers to represent or displayimages. For this reason it is necessary for the captured images to beprocessed to map the sensed colours to the desired colour system.

To correct the colours in a captured image so that they more closelymatch the colours that we would have perceived in the same scene, eachsensed colour can be mixed with appropriate percentages of the other twosensed colours. For example, an improved red signal at a point in animage may be derived from a weighted combination of the red (R), green(G) and blue (B) signals at that point.C _(r)(R)=aR+bG+cB,where C_(r)(m) gives the corrected value for a red pixel of value m.

To correct all of the colours at the point of the image, a 3×3 matrixmultiplication can be used.

$\quad\begin{matrix}{{{C( \lbrack {R\mspace{14mu} G\mspace{14mu} B} \rbrack )} = {\lbrack {R\mspace{14mu} G\mspace{14mu} B} \rbrack\mspace{14mu}\lbrack {a\mspace{14mu} d\mspace{14mu} g} \rbrack}},} \\{\lbrack {b\mspace{14mu} e\mspace{14mu} h} \rbrack{~~}} \\{\lbrack {c\mspace{14mu} f\mspace{14mu} i} \rbrack{~~}}\end{matrix}$where C(n) gives the corrected values for the RGB triplet n.

The matrix values a, b, . . . i, depend upon the characteristics of thesensor, the camera optics and the illumination of the scene, as well ason the human or computer colour system being matched. The derivation ofthese values is outside the scope of this invention.

Since the colour correction process requires all three colour values ateach pixel location, it is normally applied after demosaicing the sensordata. The correction requires considerable computation for each colourvalue: nine multiplications and six additions per pixel in each image.

It is often the case that one colour channel will have more noise thanthe other colour channels. For example, the colour sensor arrayscommonly used in still or video electronic cameras have red, green andblue filters to generate the colour image. The blue channel in the imageis often darker that the other channels, owing to the way in which theblue filters are manufactured. As a result of this, the blue channel hasworse signal-to-noise characteristics. Colour correction mixes thedifferent colour planes, and so mixes the noisy blue information intothe red and green channels. This situation is much worse for lowsaturation filters with broad spectral properties, as the colourcorrection calculations involve matrices with large off-axis componentswhich have the effect of amplifying the noise.

It is an object of the invention to provide an efficient method forperforming the colour correction step, for example when reconstructingan image from a filter mosaic.

According to a first aspect of the invention, there is provided imageprocessing method for correcting the colour of a colour image, thecolour image being composed of at least two sets of image pixels, eachset having a distinct colour value, and each pixel having a luminancevalue representing the intensity of the colour value for that pixel inthe colour image, the method comprising the steps of:

i) for each colour value, generating from the pixels for that colourvalue a low spatial frequency monochrome image composed of a set ofsmoothed image pixels;

ii) for at least one colour value, selecting a sub-set of the smoothedimage pixels;

iii) applying a colour correction function to the or each sub-set ofsmoothed image pixels in order to generate one or more correspondingsub-sets of corrected smoothed image pixels, said function including foreach colour value a contribution from corresponding smoothed imagepixels in one or more of the other sets of smoothed image pixels;

iv) completing one or more sets of corrected smoothed image pixels byinterpolating and/or extrapolating from the or each of the correctedsmoothed image pixels to generate similarly corrected smoothed imagepixels not in the original sub-set of smoothed image pixels; and

v) using the or each set of corrected smoothed image pixels to generatea corrected colour image.

Also according to a first aspect of the invention, there is provided animage processing device for correcting the colour of a colour image, thedevice comprising a processor, software, and a memory, in which thememory stores image data representative of a colour image having atleast two sets of image pixels, each set having a distinct colour value,and each pixel having a luminance value representing the intensity ofthe colour value for that pixel in the colour image, wherein theprocessor, software and memory are operable to:

a) for each colour value, generate from the pixels for that colour valuea low spatial frequency monochrome image composed of a set of smoothedimage pixels;

b) for at least one colour value, select a sub-set of the smoothed imagepixels;

c) apply a colour correction function to the or each sub-set of smoothedimage pixels in order to generate one or more corresponding sub-sets ofcorrected smoothed image pixels, said function including for each colourvalue a contribution from corresponding smoothed image pixels in one ormore of the other sets of smoothed image pixels;

d) complete one or more sets of corrected smoothed image pixels byinterpolating and/or extrapolating from the or each of the correctedsmoothed image pixels to generate similarly corrected smoothed imagepixels not in the original sub-set of smoothed image pixels; and

e) use the or each set of corrected smoothed image pixels to generate acorrected colour image.

The invention is useful in various image processing systems. Forexample, it is possible to use the invention to help de-mosaic a colourimage formed as an image mosaic from sets of interleaved pixels, eachpixel set having a different colour value. Such mosaic pixels areusually interleaved in rows and columns across the image mosaic withpixels of different colour values to form the image mosaic in such a waythat at least some rows and/or columns contain image pixels of at leasttwo colours. In the Bayer pattern, each row and column contains imagepixels of two colours.

In particular, the colour content of a colour image can be processed toremove high frequency content prior to being recombined with achromatichigh frequency content. Therefore, the method may in step v) comprisethe steps of generating from the original image pixels a high spatialfrequency luminance image; and combining the high spatial frequencyluminance images with said set(s) of corrected or uncorrected smoothedimage pixels to form the corrected full colour image.

When using the achromatic high spatial frequency image to add back inimage details, starting from an image mosaic, the high spatial frequencyluminance image is preferably generated for each colour value onlyacross pixels locations of the image mosaic for that colour value. Eachof these high spatial frequency luminance images is then combined witheach of the corresponding set(s) of corrected smoothed image pixels, orwhen there is no such corrected set, the corresponding set(s) ofsmoothed image pixels.

The intermediate smoothed image pixels may be similarly aligned alongrows and columns. Since the rate of change of colour content across thelow frequency image is reduced, the colour correction need not becomputed in full at all pixel locations. This results in a significantsavings in the required processing power and time, and associatedhardware cost. As a result, the invention is useful whenever anintermediate low spatial frequency image is generated as part of animage processing scheme.

According to a second aspect of the invention, there is provided animage processing method for correcting the colour of a colour image, theimage being composed of a plurality of image pixels and each image pixelhaving one of at least two colour values and having a luminance valuerepresenting the intensity of the colour value for that pixel in theimage, the method comprising the steps of:

I) for each colour value, generating from the pixels for that colourvalue a low spatial frequency monochrome image composed of smoothedimage pixels;

II) applying a colour correction function to the smoothed image pixelsfor one or more of colour values in order to generate one or morecorresponding corrected smoothed images, said function including for theor each colour value a contribution from corresponding smoothed imagepixels having one or more of the other colour values;

III) generating one or more high spatial frequency luminance images fromthe image; and

IV) combining the high spatial frequency luminance image(s) with the oreach of the corrected low spatial frequency monochrome images to formthe full colour image.

This approach reduces noise in the images used in the colour correction,namely the low spatial frequency monochrome images. Therefore, theeffect of noise in one or more of the channels has a correspondinglyreduced effect on the colour correction process.

As with the first aspect of the invention, said colour correction instep II) may be applied only to a sub-set of low spatial frequency imagepixels. Then, prior to step IV) said corrected low spatial frequencymonochrome image is completed by interpolating and/or extrapolating fromthe sub-set of corrected smoothed image pixels to generate similarlycorrected smoothed image pixels not in the original sub-set of correctedsmoothed image pixels.

Also according to the second aspect of the invention, there is providedan image processing device for correcting the colour of a colour image,the device comprising a processor, software, and a memory, in which thememory stores image data representative of a plurality of image pixels,each image pixel having one of at least two colour values and each pixelhaving a luminance value representing the intensity of the colour valuefor that pixel in the image, wherein the processor, software and memoryare operable to:

A) for each colour value, generate from the pixels for that colour valuea low spatial frequency monochrome image composed of smoothed imagepixels;

B) apply a colour correction function to the smoothed image pixels forone or more of colour values in order to generate one or morecorresponding corrected smoothed images, said function including for theor each colour value a contribution from corresponding smoothed imagepixels having one or more of the other colour values;

C) generate one or more high spatial frequency luminance images from theimage; and

D) combine the high spatial frequency luminance image(s) with the oreach of the corrected low spatial frequency monochrome images to formthe full colour image.

The following comments apply equally to both aspects of the invention.

Although a general colour correction scheme may involve the generationof one colour corrected smoothed image for each colour value, it maysometimes be the case that fewer or just one of the smoothed images hasto be colour corrected.

In some applications, for example reconstruction of a full colour imagefrom image mosaic data, it is preferable if the low frequency monochromeimage, generated for each colour, extends across all pixel locations ofthe original colour image.

In the case of an image mosaic, the pixels of one colour value will notoccupy all pixel locations in the original colour image. Therefore, inthe generation of the original smoothed image, it may be convenient tointerpolate between the original image pixels of at least one colourvalue to create additional image pixels of the same colour value inorder to produce a corresponding interpolated image for said colourvalue. Such an interpolation can be implemented with relatively littlecomputation, compared with the computation needed for the colourcorrection function.

In most image mosaic schemes, such as mosaics pixels arranged in theBayer pattern, there is a predominance of pixels of one colour value ascompared with the other colour values, each row and column containingimage pixels of both the predominant colour and one of the othercolours.

In a preferred embodiment of the invention, the smoothed image pixelsconsists of image pixels that lie in both alternate rows and alternatecolumns. Compared with colour correcting all smoothed image pixels, thisreduces the colour correction computation by an amount that approaches75% for large images. At the same time, each interpolated orextrapolated smoothed image pixel is adjacent one of the colourcorrected smoothed image pixels.

The invention will now be described in further detail, by way of examplewith reference to the following drawings:

FIG. 1 shows schematically some of the internal components of aconventional electronic camera, including a colour sensor array havingred, green and blue (RGB) imaging elements;

FIG. 2 shows an example of original black and white text to be imaged bya colour sensor array such as that in FIG. 1;

FIG. 3 shows the text when captured as an RGB mosaic image;

FIG. 4 shows the arrangement of red, green and blue pixels in the imagemosaic of FIG. 3;

FIG. 5 shows schematically how the red, green and blue pixels areprocessed in a method according to the invention to yield colourcorrected intermediate smoothed images, one for each colour value, andhow these corrected images are then used to generate a de-mosaiced andcolour corrected full colour image;

FIG. 6 is a circuit schematic diagram for a device according to theinvention for de-mosaicing an image mosaic to form a full colour image,the device comprising a processor, software, and a memory; and

FIG. 7 is a flow chart illustrating a preferred embodiment of the methodaccording to the invention for de-mosaicing an image mosaic to yield afull colour image.

FIG. 1 shows one example of a consumer imaging device, here a hand-helddigital camera 1. Such cameras have a colour image sensor 2 having atwo-dimensional regular array of imaging elements or pixels 4. A typicalconsumer sensor array may have up to 4 Megapixels resolution, arrangedin a rectangular array 2500 pixels wide and 1600 pixels high.

The imaging elements are sensitive to light across a wide spectrum ofcolours, and so the sensor array 2 is overlain by a mosaic-like patternof colour filters 6. There are usually only three such colours, red (R),green (G) and blue (B), (RGB) and the colours are usually interleaved ina repeating pattern across the sensor array 2. Thus, the array elements4 under each colour of filter 6 are sensitive only to light withwavelengths passed by each corresponding filter 6.

Many filter patterns exist, but the most common is the Bayer filterpattern. This consists of pixels with colour filters arranged in arectangular grid pattern as set out below:

$\quad\begin{matrix}G & R & G & R & . & . & . & G & R & G & R \\B & G & B & G & . & . & . & B & G & B & G \\G & R & G & R & . & . & . & G & R & G & R \\. & . & . & . & . & . & . & . & . & . & . \\B & G & B & G & . & . & . & B & G & B & G \\G & R & G & R & . & . & . & G & R & G & R \\B & G & B & G & . & . & . & B & G & B & G\end{matrix}$where R, G and B represent red, green and blue colour filtersrespectively. For the Bayer pattern, there is a preponderance of greenpixels in the sensor array, with these contributing half of the fullsensor resolution, while the red and blue pixels each contribute onequarter of the resolution.

FIG. 2 shows an example of original black and white text 10, consistingof the symbols “• UP”, to be imaged by the digital camera 1. The text 10is part of a larger document (not shown) that is to be imaged in adesktop document imaging application. FIG. 3 shows how the text 10 isimaged as an RGB mosaic 12 within a small portion of the sensor array 2consisting of 35 pixels in a horizontal direction and 29 pixels in avertical direction.

The Bayer pattern can be seen most clearly in FIG. 4, which shows forthe three colour values red 16, green 17 and blue 19, that there aretwice as many green pixels 14 as red pixels 13 or blue pixels 15. Theoriginal text 10 is visible as different luminance levels of the pixels13,14,15.

FIG. 5 shows schematically how the image mosaic 12 is processed to yielda colour corrected and de-mosaiced full colour image 20. First, for eachcolour value 16,17,18, a smoothed image 33,34,35 is formed. Then, foreach of the three colour values 16,17,18, a difference is taken 26,27,28between the luminance level of each individual pixel 13,14,15 for thatparticular colour value 16,17,18, and a corresponding point of thesmoothed image of the same colour value. This difference 26,27,28 isused to generate an achromatic high frequency image which when combinedwith similar differences for the other two colour values results in acomposite achromatic high frequency image 30 that extends across allpixels locations in the original RGB image mosaic 12.

Therefore, the resulting composite image 30 is a black and white highfrequency version of the original RGB image 12. Most conveniently, thehigh frequency image 30 consists of three sets of high frequency imagepixels 43,44,45 at locations in the composite image 30 that correspondwith the locations of corresponding sets of pixels 13,14,15 in theoriginal RGB mosaic image 12. As can be seen in FIG. 5, these pixels43,44,45 have different luminance values.

Each of the smoothed monochrome images 33,34,35 is formed bytwo-dimensional interpolation combined with low-pass filtering (examplesof spatial filters including low pass or smoothing filters are given inDigital Image Processing, by Gonzalez and Woods, pages 189 to 201,Addison & Wesley, 1992). Here, the smoothed images 33,34,35 are formedindividually for each of the three colour values 16,17,18 using bilinearinterpolation as a pre-processing step. All three smoothed images33,34,35 then extend across locations corresponding with all elements ofthe RGB mosaic pattern 12.

One or more of the three sets of low spatial frequency image pixels33,34,35 for the colour values 16,17,18 are then processed as follows inorder to generate up to three corresponding colour corrected low spatialfrequency image pixels 53,54,55. FIG. 5 shows the general case where allthree sets of original smoothed image pixels 33,34,35 are processed tomake a colour correction. The colour correction need not change theactual colour of the smoothed images, but will usually be a correctionto intensity values in up to three of the sets of smoothed pixels.

For example, as described in the introduction, an improved green signalat a point in an image may be derived from a weighted combination of thered (R), green (G) and blue (B) signals at that point.C _(g)(G)=aR+bG+cB,where C_(g)(m) gives the corrected value for a green pixel of value m.

To correct all of the colours at the point of the image, a 3×3 matrixmultiplication can be used:

$\quad\begin{matrix}{{{C( \lbrack {R\mspace{14mu} G\mspace{14mu} B} \rbrack )} = {\lbrack {R\mspace{14mu} G\mspace{14mu} B} \rbrack\mspace{14mu}\lbrack {a\mspace{14mu} d\mspace{14mu} g} \rbrack}},} \\{\lbrack {b\mspace{14mu} e\mspace{14mu} h} \rbrack{~~}} \\{\lbrack {c\mspace{14mu} f\mspace{14mu} i} \rbrack{~~}}\end{matrix}$where C(n) gives the corrected values for the RGB triplet n.

Although a colour correction may be needed for all three colour values16,17,18, for simplicity, the details of just one colour correction forthe green colour value 17 are described below. In order to reduce thecomputational requirement, the invention proposes performing the colourcorrection only on a sub-set 63,64,65 of the complete set of smoothedimage pixels 33,34,35.

Consider now a 3×3 block of uncorrected low spatial frequency smoothedgreen image pixels, denoted by the values G1 to G9:

$\quad\begin{matrix}{G1} & {G2} & {G3} \\{G4} & {G5} & {G6} \\{G7} & {G8} & {G9}\end{matrix}$

The corresponding colour corrected low spatial frequency green values g1to g9 can be represented by:

$\quad\begin{matrix}{g1} & {g2} & {g3} \\{g4} & {g5} & {g6} \\{g7} & {g8} & {g9}\end{matrix}$

A sub-set is of the smoothed image pixels G1–9 is selected, consistingof the corner values G1, G3, G7 and G9. The colour correctioncalculation is performed only for these four image pixels, with theother image pixels being interpolated, as follows:

$\begin{matrix}{g1} & {g2} & {g3} \\{g4} & {g5} & {g6} \\{g7} & {g8} & {g9}\end{matrix} = \begin{matrix}{C_{g}({G1})} & {( {{g1} + {g3}} )/2} & {C_{g}({G3})} \\{( {{g1} + {g7}} )/2} & {( {{g1} + {g3} + {g7} + {g9}} )/4} & {( {{g3} + {g9}} )/2} \\{C_{g}({G7})} & {( {{g7} + {g9}} )/2} & {C_{g}({G9})}\end{matrix}$

There are, of course, many other possible sub-sets which may beselected, and many other interpolation or extrapolation schemes forcalculating corrected image pixels not in the selected sub-set. Thisexample, however, in which the selected smoothed image pixels come fromboth alternate rows and alternate columns, gives excellent results witha significant savings in the required computation.

Methods such as the one described herein require far less computationthan traditional colour correction methods. For example, the describedmethod requires approximately one-quarter of the computation with littleor no detectable loss in image quality.

Furthermore, because the colour correction is performed on low spatialfrequency pixels in which noise in any of the colour channels issmoothed out, the colour correction does not mix noise between colourplanes or increase the amount of noise in the colour corrected image.

After colour correction, the image can be demosaiced as follows. Foreach high frequency pixel 43,44,45, the achromatic high frequencyluminance value is added 50 to a corresponding portion of each of thethree colour corrected smoothed images 53,54,55, which results in ade-mosaiced full colour image 20. In the case that one or two of thecolour values 16,17,18 do not require colour correction, then at thisstage in the computation for that uncorrected colour value, the originaluncorrected smoothed set of image pixels 33,34,35 is combined with thecorresponding high frequency pixels 43,44,45.

This method has the advantage of being relatively easy to compute inelectronic hardware, while still giving good reconstructed image qualityfor colour corrected images.

Rather than interpolating the individual colour planes to fullresolution, prior to smoothing, only to then sample the smoothedversions for colour correction it is possible to reduce the computationrequired still further by reducing the number of mosaic pixel locationsat which the mosaic pattern is spatially filtered to produce the lowspatial resolution images. This results in smoothed images with lowerspatial frequencies than might otherwise be the case (unless the degreeof smoothing is itself reduced to accommodate the change in spatialresolution), but at no great ultimate loss in information content, forthe reason that image details are reintroduced by adding the highfrequencies as described above. Preferably, smoothed images are computedonly for a subset of the green pixels 14, for example as shown in FIG. 4those green pixels in columns 54 (or alternatively rows) having bothgreen pixels 14 and red pixels 13. For the Bayer pattern, this requiresone quarter of the computation, required for smoothing, while the imagequality remains almost constant.

For red and blue pixels, 13 and 15 respectively, this amounts toavoiding the initial interpolation stage and operating the smoothing onimages formed from the raw red and blue pixels alone.

Therefore, each of the high spatial frequency images 44,45,46 is formedfor each of the colour values 16,17,18 from the difference 26,27,28between the luminance values of the image mosaic pixels 16,17,18 forthat colour value and corresponding portions of the smoothed monochromeimage 33,34,35 for that same colour value 16,17,18.

In other words, the high frequency component of each mosaic pixel isgiven by subtracting the mosaic value from the corresponding location ofa smoothed version of the image for the same colour value.

The colour corrected and de-mosaiced full colour image 20 is then formedfor each of the colour values 16,17,18 by summing each of the highspatial frequency images 30 with corresponding portions of the colourcorrected smoothed monochrome images 53,54,55 for that colour value.

The process described above may be readily implemented in hardware,illustrated in block schematic form in FIG. 6, and illustrated in theflowchart of FIG. 7.

A shutter release mechanism 8 when activated by a user sends a signal 71to a microprocessor unit 72, which may include a digital signalprocessor (DSP). The microprocessor then sends an initiation signal 73to a timing generator 74, whereupon the timing generator sends a triggersignal 75 to an electronic image sensor unit 76.

The sensor 76 consists of an imaging area 77 consisting of an array ofsensing elements (typically either of a photogate or alternativelyphotodiode construction) and a serial readout register 78 from where ananalogue signal 79 is generated via an amplifier 80. This signal 79 isgenerated upon receipt by the sensor unit 76 of the trigger signal 75.

The amplified analogue signal 79 is converted to a digital signal 81 byan A/D unit 82. The resulting raw digital image data is storedtemporarily in a volatile memory 84.

Image processing according to the present invention can then beperformed by the microprocessor unit 72. The microprocessor may includeadditional DSP capability in the form of specialised block of hardwareto carry out specific functions or an additional more general DSPco-processor.

The processing itself may be performed according to the steps outlinedin the flow-chart of FIG. 7. These include a pre-processing stage 92,which may typically include correction of the OECF (opto-electronicconversion function) of the sensor and white-balancing to compensate forvariations in illumination. Following the colour correction andde-mosaicing stage 94 described above, a subsequent post-processingstage 96 may include exposure correction (which can also be accomplishedat the pre-processing stage) and transformation to a standard colourspace such as sRGB (as described in IEC 61966-2-1). Finally thereconstructed RGB image data can be-compressed 98 and stored in longterm memory 88 using a standard image compression scheme such as theubiquitous JPEG scheme.

Additionally a display device 90 may be incorporated into the design.Images can be displayed live to facilitate view-finding or reviewed fromlong term memory requiring an additional decompress processing stage100.

Although a preferred embodiment of the invention has been described withreference to the Bayer pattern of image pixels, the invention isapplicable to cases where not all rows and/or columns contain imagepixels of at least two colours. For example, some mosaics have puregreen rows and columns interleaved with red/blue rows or columns. Theinvention is equally applicable to such image mosaics.

It is not strictly necessary to store the whole raw image frame involatile memory. The image processing can be performed on the fly, thusrequiring only as much memory as is necessary to perform the imagingpipeline. So after the first few rows of image data have been read fromthe sensor into memory it is possible to generate compressed image datafor the start of the image and begin storing these in long term memory.This results from the fact that all processes are essentially local andoperate only on a limited area of the image.

In other words, although the “images” constructed at each stage of theprocess could be complete sets of data that extend across the entireimage, in practice this adds cost in terms of memory and possiblethroughput. Therefore, the “images” used at each stage of the processwill in general be created piecemeal, with the process operatinglocally. In the limit all the computation may be carried out for asingle pixel from the pixels in its neighbourhood.

The invention described above maintains image detail from all of thesensor elements while reconstructing a full RGB image. It can beincorporated into any scheme that is based upon the combination of anachromatic high frequency image with low frequency red, green and blueimages (or other primary colour combinations). Preferably it isincorporated into an additive reconstruction scheme whereby theachromatic high frequencies are added to each of the low frequencycolour channels.

The method described above for colour correcting and reconstructing afull colour image from image mosaic data is computationally attractiveand utilises all three colour values of the mosaic to produce highresolution colour images. The method also significantly reduces theeffect of noise in the original colour channels on the colour correctionprocess.

In the colour correction computation, a good compromise between speedand quality is achieved when the colour correction is computed foralternate pixel rows and columns of the low frequency image, with theresults being interpolated to the other nearby pixels.

Although particularly useful in the context of demosaicing the imagedata from a colour mosaic sensor, this invention can be applied in othercontexts where a colour correction or colour space conversion isrequired, even if the image to be corrected already has red, green andblue values for each pixel. For example, an image to be mapped into adifferent colour space for printing could be converted into three lowfrequency colour separations plus one or more high frequency components.The low frequency separations could be colour corrected, possibly at areduced spatial resolution with subsequent interpolation back to theoriginal resolution, prior to recombination with the high frequencycomponent(s). Thus the conversion could be performed with reducedcomputation and with potential improvements in image quality.

Furthermore, although in the above example, the image data is of animage mosaic, the invention can be applied to cases where each locationin the original colour image has overlapping or coincident pixels ofmultiple colour values. Such images can be generated by imaging systemshaving three CCD sensors, by flatbed colour scanners, or other imagingsystems in which the colour image is formed from sets of separate,registered RGB images.

The invention therefore provides an efficient method with reducedsensitivity to image noise for colour correcting and reconstructing ahigh quality image, with the full sensor resolution in each of the red,green and blue colour channels.

1. An image processing method of correcting the colour of a colourimage, the colour image including at least two sets of image pixels,each set having a distinct colour value, and each pixel having aluminance value representing the intensity of the colour value for thatpixel in the colour image, the method comprising the steps of: i) foreach colour value, generating from the pixels for that colour value alow spatial frequency monochrome image including a set of smoothed imagepixels; ii) for at least one colour value, selecting a sub-set of thesmoothed image pixels; iii) generating one or more correspondingsub-sets of corrected smoothed image pixels by applying a colourcorrection function to the or each sub-set of smoothed image pixels,said function including for each colour value a contribution fromcorresponding smoothed image pixels in one or more of the other sets ofsmoothed image pixels; iv) completing one or more sets of correctedsmoothed image pixels by at least one of interpolating and extrapolatingfrom the or each of the corrected smoothed image pixels to generatesimilarly corrected smoothed image pixels not in the original sub-set ofsmoothed image pixels; and v) generating a corrected colour image byusing the or each set of corrected smoothed image pixels.
 2. A method asclaimed in claim 1, in which step v) comprises the steps of generatingfrom the original image pixels a high spatial frequency luminance image;and combining the high spatial frequency luminance images with saidset(s) of corrected or uncorrected smoothed image pixels to form thecorrected full colour image.
 3. A method as claimed in claim 2, in whichthe colour image is an image mosaic, and for each colour value,generating a high spatial frequency luminance image that extends onlyacross pixels locations of the image mosaic for that colour value, andcombining each of said high spatial frequency luminance images with eachof the corresponding set(s) of corrected smoothed image pixels, or whenthere is no such corrected set, the corresponding set(s) of smoothedimage pixels.
 4. An image processing method of correcting the colour ofa colour image, the image including a plurality of image pixels and eachimage pixel having one or at least two colour values and having aluminance value representing the intensity of the colour value for thatpixel in the image, the method comprising the steps of: I) for eachcolour value, generating from the pixels for that colour value a lowspatial frequency monochrome image including smoothed image pixels; II)generating one or more corresponding corrected smoothed images byapplying a colour correction function to the smoothed image pixels forone or more of the colour values, said function including for the oreach colour value a contribution from corresponding smoothed imagepixels having one or more of the other colour values; III) generatingone or more high spatial frequency luminance images from the image; andIV) combining the high spatial frequency luminance image(s) with the oreach of the corrected low spatial frequency monochrome images to formthe full colour image.
 5. An image processing method as claimed in claim4 in which: V) in step II) said colour correction is applied only to asub-set of low spatial frequency image pixels; and VI) prior to step IV)said corrected low spatial frequency monochrome image is completed byinterpolating and/or extrapolating from the sub-set of correctedsmoothed image pixels to generate similarly corrected smoothed imagepixels not in the original sub-set of corrected smoothed image pixels.6. An image processing method as claimed in claim 1, in which the colourimage is an image mosaic, the pixels of each colour value beinginterleaved in rows and columns across the image mosaic with pixels ofdifferent colour values to form the image mosaic in such a way that oneor more of at least one of the rows and columns include image pixels ofat least two colours.
 7. An image processing method as claimed in claim6, in which there is a predominance of pixels of one colour value ascompared with the other colour values, each row and column includingimage pixels of both the predominant colour and one of the othercolours.
 8. An image processing method as claimed in claim 7, in whichsaid smoothed image pixels consist of image pixels that lie in bothalternate rows and alternate columns.
 9. An image processing method asclaimed in claim 4, in which the colour image is an image mosaic, thepixels of each colour value being interleaved in rows and columns acrossthe image mosaic with pixels of different colour values to form theimage mosaic in such a way that a plurality of at least one of the rowsand columns includes image pixels of at least two colours.
 10. An imageprocessing method as claimed in claim 9, in which there is apredominance of pixels of one colour value as compared with the othercolour values, each row and column including image pixels of both thepredominant colour and one of the other colours.
 11. An image processingmethod as claimed in claim 10, in which said smoothed image pixelsinclude image pixels that lie in both alternate rows and alternatecolumns.
 12. A method as claimed in claim 1, in which the low frequencymonochrome image extends across all pixel locations of the originalcolour image.
 13. A method as claimed in claim 4, in which the lowfrequency monochrome image extends across all pixel locations of theoriginal colour image.
 14. An image processing method as claimed inclaim 1, in which the colour values comprise red, green and blue, therebeing a preponderance of green pixels in the image mosaic.
 15. An imageprocessing method as claimed in claim 4, in which the colour valuescomprise red, green and blue, there being a preponderance of greenpixels in the image mosaic.
 16. An image processing device forcorrecting the colour of a colour image, the device comprising aprocessor, software, and a memory, in which the memory is arranged tostore image data representative of a colour image having at least twosets of image pixels, each set having a distinct colour value, and eachpixel having a luminance value representing the intensity of the colourvalue for that pixel in the colour image, wherein the processor,software and memory are operable to: a) for each colour value, generatefrom the pixels for that colour value a low spatial frequency monochromeimage consisting of a set of smoothed image pixels; b) for at least onecolour value, select a sub-set of the smoothed image pixels; c) apply acolour correction function to the or each subset of smoothed imagepixels in order to generate one or more corresponding sub-sets ofcorrected smoothed image pixels, said function including for each colourvalue a contribution from corresponding smoothed image pixels in one ormore of the other sets of smoothed image pixels; d) complete one or moresets of corrected smoothed image pixels by at least one of interpolatingand extrapolating from the or each of the corrected smoothed imagepixels to generate similarly corrected smoothed image pixels not in theoriginal sub-set of smoothed image pixels; and e) use the or each set ofcorrected smoothed image pixels to generate a corrected colour image.17. An image processing device for correcting the colour of a colourimage, the device comprising a processor, software, and a memory, inwhich the memory is arranged to store image data representative of aplurality of image pixels, each image pixel having one of at least twocolour values and each pixel having a luminance value representing theintensity of the colour value for that pixel in the image, wherein theprocessor, software and memory are operable to: A) for each colourvalue, generate from the pixels for that colour value a low spatialfrequency monochrome image consisting of smoothed image pixels; B) applya colour correction function to the smoothed image pixels for one ormore of colour values in order to generate one or more correspondingcorrected smoothed images, said function including for the or eachcolour value a contribution from corresponding smoothed image pixelshaving one or more of the other colour values; C) generate one or morehigh spatial frequency luminance images from the image; and D) combinethe high spatial frequency luminance image(s) with the or each of thecorrected low spatial frequency monochrome images to form the fullcolour image.
 18. The method as claimed in claim 1, wherein the sub-setselecting step for the at least one colour value is for only that colourvalue and the completing step is for said at least one colour value andthe corrected smoothed image pixels are for that colour value.
 19. Themethod as claimed in claim 4, wherein the function for each particularcolour value results from the contribution having only the particularcontribution.
 20. The device as claimed in claim 16, wherein theselected sub-set for a particular at least one colour value is of onlythe particular colour value and function for each particular colourvalue results from the contribution having only the particularcontribution.
 21. An image processing method of correcting the colour ofa colour image, the image including a plurality of image pixels, eachimage pixel having at least one of at least two colour values and havingat least one value representing the respective intensity of each colourvalue for that pixel in the image, the method comprising the steps of:I) for each colour value, generating from the pixels for that colourvalue a noise-reduced image including filtered image pixels; II)generating one or more corresponding corrected filtered images byapplying a colour correction function to the filtered image pixels forone or more of the colour values, the function including, for eachcolour value, a contribution from corresponding filtered image pixelshaving one or more of the other colour values; III) generating one ormore high speed spatial frequency luminance images from the images; andIV) forming a corrected colour image by combining the one or more highspatial frequency luminance images with the one or more correctedfiltered images.
 22. An image processing method as claimed in claim 1,wherein the one or more corrected filtered images are formed by applyinga two-dimensional smoothing function of all pixel locations.
 23. Animage processing method as claimed in claim 21, wherein thenoise-reduced image is formed by applying a two-dimensional smoothingfunction at all pixel locations.
 24. Apparatus for correcting the colourof a colour image, the image including a plurality of image pixels, eachimage pixel having at least one of at least two colour values and havingat least one value representing the respective intensity of each colourvalue for that pixel in the image, the apparatus including a processorfor performing the steps of claim
 21. 25. The apparatus as claimed inclaim 24, wherein the processor is arranged to form the noise-reducedimage by applying a two-dimensional smoothing function at all pixellocations.